Information retrieval system evaluation method, device and storage medium

ABSTRACT

The present disclosure discloses an information retrieval system evaluation method, device and storage medium. A ratio between the sum of all the related object parameters of a keyword in keyword in an evaluation retrieval result set and the sum of all the related object parameters of the keyword in keyword in a retrieval result set is used to compute a recall rate of an information retrieval system. And the recall rate is introduced to evaluate the information retrieval system, thereby enhancing accuracy of quantitative evaluation of the information retrieval system, and improving the automation degree of evaluation.

This application claims priority to and is a continuation application ofPCT/CN2013/090906, filed on Dec. 30, 2013 and entitled “INFORMATIONRETRIEVAL SYSTEM EVALUATION METHOD, DEVICE AND STORAGE MEDIUM”, whichclaims priority to Chinese Patent Application No. 201310084139.4, filedwith the Chinese Patent Office on Mar. 15, 2013 by Tencent technology(shenzhen) Co., Ltd. and entitled “INFORMATION RETRIEVAL SYSTEMEVALUATION METHOD AND DEVICE”, which are incorporated herein referencein their entirety.

TECHNICAL FIELD

The present disclosure relates to the field of communicationstechnologies, and in particular, to an information retrieval systemevaluation method, device and storage medium.

BACKGROUND

This section provides background information related to the presentdisclosure which is not necessarily prior art.

An information retrieval system forms a core technology of an existingInternet application, and the information retrieval system may be usedto retrieve a web page, a music file, a video file and an image file inthe whole Internet, and may also be used to retrieve information and asubject in a single website or database, and the quality of a retrievalresult thereof has great influence to the usage experience of anInternet application. An evaluation method of an existing informationretrieval system often relies on manual evaluation, and requirespersonnel to participate much work.

And, the evaluation mainly focuses on information retrieval such asaddressing retrieval. For an information retrieval system of anothertype, the evaluation is weak in applicability, bad in accuracy, andcannot reflect the retrieval performance of the information retrievalsystem.

SUMMARY

Therefore, it is an object of the embodiments of the present disclosureto provide an information retrieval system evaluation method, device andstorage medium to solve one or more problems set forth above and otherproblems.

The embodiments of the present disclosure provide an informationretrieval system evaluation method, which includes:

obtaining behavior data sample reported by an information retrievalsystem within a pre-determined period;

obtaining a sample retrieval keyword set and a sample retrieval resultcorresponding to each sample retrieval keyword according to the behaviordata sample;

invoking the information retrieval system to perform evaluationretrieval on a pre-determined evaluation keyword set, computing a recallrate and a correctness percentage corresponding to each keyword in theevaluation keyword set according to an evaluation retrieval result andthe sample retrieval result; and

computing an evaluation indicator of the information retrieval systemaccording to the recall rate and the correctness percentage, where

the recall rate is a ratio between the sum of all related objectparameters in the evaluation retrieval result corresponding to a keywordand the sum of all the related object parameters in the sample retrievalresult corresponding to the keyword;

the correctness percentage is computed according to the number of therelated objects and the number of the non-related objects in theevaluation retrieval result corresponding to the keyword; and

the related object corresponding to the keyword is an object where auser performs operation in the sample retrieval result corresponding tothe keyword; the non-related object corresponding to the keyword is anobject where a user does not performs operation in the sample retrievalresult corresponding to the keyword.

The embodiments of the present disclosure further disclose aninformation retrieval system evaluation device, comprising: one or moreprocessors, memory and one or more program units stored in the memoryand to be executed by the one or more processors, the one or moreprogram units comprising: a behavior data collecting unit, an analyzingunit, an evaluation retrieval unit and an evaluation indicator computingunit; wherein

the behavior data collecting unit is configured to obtain behavior datasample reported by an information retrieval system within apre-determined period;

the analyzing unit is configured to obtain a sample retrieval keywordset and a sample retrieval result corresponding to each sample retrievalkeyword according to the behavior data sample;

the evaluation retrieval unit is configured to invoke the informationretrieval system to perform evaluation retrieval on a pre-determinedevaluation keyword set, compute a recall rate and a correctnesspercentage corresponding to each keyword in the evaluation keyword setaccording to an evaluation retrieval result and the sample retrievalresult; and

the evaluation indicator computing unit is configured to compute anevaluation indicator of the information retrieval system according tothe recall rate and the correctness percentage, where

the recall rate is a ratio between the sum of all related objectparameters in the evaluation retrieval result corresponding to a keywordand the sum of all the related object parameters in the sample retrievalresult corresponding to the keyword;

the correctness percentage is computed according to the number of therelated objects in the evaluation retrieval result corresponding to thekeyword and the number of the non-related objects in the retrievalresult sub-set; and

the related object corresponding to the keyword is an object where auser performs operation in the sample retrieval result corresponding tothe keyword; the non-related object corresponding to the keyword is anobject where a user does not performs operation in the sample retrievalresult corresponding to the keyword.

The embodiments of the present disclosure further disclose a storagemedium containing a computer executable instruction, the computerexecutable instruction is used to perform an information retrievalsystem evaluation method when executed by a compute processor, where themethod includes the following steps:

obtaining behavior data sample reported by an information retrievalsystem within a pre-determined period;

obtaining a sample retrieval keyword set and a sample retrieval resultcorresponding to each sample retrieval keyword according to the behaviordata sample;

invoking an information retrieval system to perform evaluation retrievalon a pre-determined evaluation keyword set, computing a recall rate anda correctness percentage corresponding to each keyword in the evaluationkeyword set according to an evaluation retrieval result and the sampleretrieval result; and

computing an evaluation indicator of the information retrieval systemaccording to the recall rate and the correctness percentage, where

the recall rate is a ratio between the sum of all the related objectparameters in an evaluation retrieval result corresponding to a keywordand the sum of all the related object parameters in a sample retrievalresult corresponding to the keyword;

the correctness percentage is computed according to the number of therelated objects and the number of the non-related objects in theevaluation retrieval result corresponding to the keyword; and

the related object corresponding to the keyword is an object where auser performs operation in the sample retrieval result corresponding tothe keyword; the non-related object corresponding to the keyword is anobject where a user does not performs operation in the sample retrievalresult corresponding to the keyword.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic flow chart of an information retrieval systemevaluation method in a first embodiment of the present disclosure;

FIG. 2 is a schematic structural diagram of an information retrievalsystem evaluation device in a second embodiment of the presentdisclosure; and

FIG. 3 is a schematic diagram of an implementation environment of aninformation retrieval system evaluation method in embodiments of thepresent disclosure.

DESCRIPTION OF EMBODIMENTS

Technical solutions in the present disclosure are described in detailwith reference to the accompanying drawings and the embodiments.

FIG. 1 is a schematic flow chart of an information retrieval systemevaluation method in a first embodiment of the present disclosure. Themethod in the present disclosure may implement information retrievalsystem evaluation through an evaluation electronic device communicatedwith the information retrieval system. As shown in FIG. 3, theevaluation electronic device 31 may include various storage medium 312,a system memory 313, a processor 314 and an input and output device 315connected through a system bus 311. The storage medium 312 may bemagnetic disk, an optical disk, a read-only memory (Read-Only Memory,ROM, a random access memory (Random Access Memory, RAM), or any othermedium having a storage function.

The evaluation electronic device 31 is in communication connection withthe information retrieval system 32, the communication may be a localarea network (LAN) and a wide area network (WAN), and may further beanother network. Networking of this kind is based on a computer network,Intranet, Internet within an office of an enterprise. The evaluationelectronic device 31 may access the information retrieval system 32through communication connection, and obtain required data.

As shown in FIG. 1, the method includes the following.

Step 110: The evaluation electronic device 31 obtains behavior datasample reported by the information retrieval system 32 within apre-determined period.

Specifically, the evaluation electronic device 31 obtains relatedparameters of a user using the information retrieval system 32 andreported by an interaction module, a searching module and any anothermodule (such as a downloading module or another running module) relatedto subsequent operations in the information retrieval system 32 within apre-determined period (for example, one month preferably or a longerperiod), including a keyword used for retrieval within a certainpre-determined period, an retrieval result, an arrangement position ofeach object in the retrieval result listed in the retrieval result, andan operation such as whether a user clicks the retrieval result.

In the embodiments of the present disclosure, the retrieval resultrefers to a retrieved object list obtained through the informationretrieval system 32, and the object refers to specific retrieval targetof the information retrieval system 32, and may be a web page, adocument, a subject, an image, or files of other type.

Step 120: The evaluation electronic device 31 obtains a sample retrievalkeyword set and a sample retrieval result corresponding to each sampleretrieval keyword according to the behavior data sample.

In this step, the evaluation electronic device 31 analyzes and acquiresa sample retrieval keyword set and a retrieval result corresponding toeach sample retrieval keyword according to the behavior data sample,that is, a sample retrieval result, and the evaluation electronic device31 determines a related object and a non-related object of each sampleretrieval keyword at the same time. In the embodiment of the presentdisclosure, the sample retrieval result corresponding to each sampleretrieval keyword may be formed by a sample retrieval sub-result, whereeach sample retrieval sub-result specifically is a result obtainedthrough retrieving the keyword at different time within a pre-determinedperiod.

If in a retrieval result of retrieving a specified keyword contained inthe obtained behavior data sample, the user performs a further operation(such as click) on a specified object in the retrieval result, theobject is the related object in the specified keyword, that is, therelated object of the keyword is an object where the user performs anoperation in the sample retrieval result of the keyword.Correspondingly, if in all the retrieval results to the specifiedkeyword, the user does not perform any operation on the specified objectof all retrieval results, the object is a non-related object of thespecified keyword, that is, the non-related object of the keyword is anobject where the user does not perform a further operation in the sampleretrieval result of the keyword.

At the same time, in a preferred implementation manner of thisembodiment, the related object parameters of all related objectscorresponding to the keyword may further be determined according to thesample retrieval result of the keyword in this step, and is used torepresent dependency between the related object and the correspondingkeyword.

In another preferred implementation manner of this embodiment, if theuser may perform two different further operations on the object in theretrieval result, the related object parameter of each related objectmay be computed according to the following formula:

${score} = {{\frac{a}{ExposeCnt}{\sum\limits_{x = 1}^{FCnt}\; {FirstScore}_{x}}} + {\frac{b}{ExposeCnt}{\sum\limits_{y = 1}^{SCnt}\; {SecondScore}_{y}}} + \sigma}$

ExposeCnt is the total number of times that the related object appearsin a sample retrieval result of a corresponding keyword, FCnt is thetotal number of times that a user performs first operation on therelated object appearing in a sample retrieval result of a correspondingkeyword, SCnt is the total number of times that a user performs secondoperation on the related object appearing in a sample retrieval resultof a corresponding keyword, a is a first operation weight coefficient, bis a second operation weight coefficient, and σ is an offsetcoefficient. The evaluation electronic device 31 may adjust an influencefrom different operations to the dependency through the first operationweight coefficient and the second operation weight coefficient. At thesame time, the evaluation electronic device 31 may adjust the relatedobject parameter value through setting the offset coefficient accordingto the actual condition of the information retrieval system 32.

FirstScorex is a dependency coefficient when the related object appearsin the sample retrieval result of the corresponding keyword for an xthtime and is performed with the first operation, wherein the dependencycoefficient is computed according to an order of the related object in asample retrieval sub-result obtained in the retrieval; SecondScorey is adependency coefficient when the related object appears in the sampleretrieval result of the corresponding keyword for a yth time and isperformed with the second operation, wherein the dependency coefficientis computed according to an order of the related object in a sampleretrieval sub-result obtained in the retrieval. The closer is the rankof the related object in sample retrieval sub-result at the retrieval tothe end, the higher the dependency coefficient is, because it indicatesthat the object is a true object of interest to the user since itreceives further operation from the user despite of its bad rank.Besides, FirstScorex and SecondScorey are relevant to the rank conditionof each object in the sample retrieval sub-result obtained after eachretrieval, and therefore, FirstScorex and SecondScorey change with thechange of the sample retrieval sub-result obtained after retrieval. Itcan be obtained from the foregoing formula that, the greater the numberof times that the first operation and/or the second operation isperformed on the related object of the keyword, the greater thedependency between the related object and the keyword is, so it is withthe related object parameter.

In the present implementation manner, the first operation and the secondoperation may be different operations. For example, when the retrievedobject is a document or a web page, the first operation is clicking forviewing, and the second operation is downloading; when the retrievedobject is a music file, the first operation is clicking a listeninglink, and the second operation is downloading; when the retrieved objectis an object, the first operation may be clicking a viewing link, andthe second operation is clicking an ordering link or performing anordering operation.

In the present implementation manner, when a dependency between arelated object and a corresponding keyword is computed, considering thecontribution of several different subsequent operations to thedependency, the evaluation electronic device 31 performs multi-latitudemodeling, which better complies with the demand of an existinginformation retrieval system, and particularly complies with the demandof an information retrieval system of a non-addressing retrieval type.

Step 130: The evaluation electronic device 31 invokes the informationretrieval system 32 to perform evaluation retrieval on a pre-determinedevaluation keyword set, computes a recall rate and a correctnesspercentage corresponding to each keyword in the evaluation keyword setaccording to an evaluation retrieval result and the sample retrievalresult.

The evaluation electronic device 31 may perform evaluation afterreceiving the behavior data sample and obtaining the sample retrievalresult of each sample keyword in a sample keyword set by analyzing thesample. In this embodiment, the evaluation electronic device 31 mayfirstly perform evaluation retrieval on the pre-determined evaluationkeyword set through automatically invoking the information retrievalsystem 32, so as to obtain a group of evaluation retrieval resultscorresponding to each evaluation keyword in the evaluation keyword set.The evaluation keyword set is a subset obtained according to a samplekeyword set. And then a recall rate and a correctness percentagecorresponding to each evaluation keyword are computed according to theevaluation retrieval result and the previously obtained sample retrievalresult.

The recall rate is a ratio between the sum of all the related objectparameters in the evaluation retrieval result corresponding to a keywordand the sum of all the related object parameters in the sample retrievalresult corresponding to the keyword. The ratio represents the recallrate of the evaluation retrieval result relative to the sample retrievalresult.

In a preferable implementation manner of this embodiment, after therecall rate of each keyword is computed, an arithmetic mean value of therecall rates of all keywords is also computed. Specifically, thearithmetic mean value of the recall rates of all keywords may becomputed according to the following formula:

${R = {\frac{1}{n}{\sum\limits_{k = 1}^{n}\; R_{k}}}},{R_{k} = \frac{{HitScore}_{k}}{{TotalScore}_{k}}}$

Where, n is the number of keywords in the evaluation keyword set, Rk isa recall rate of a Kth evaluation keyword, HitScorek is the sum of allthe related object parameters in an evaluation retrieval resultcorresponding to a Kth evaluation keyword, and TotalScorek is the sum ofall the related object parameters in a sample retrieval resultcorresponding to a Kth evaluation keyword. The related object parameteris computed in step 120 and is used to represent the dependency betweenthe related object and the corresponding keyword.

In this embodiment, the related object parameter is introduced torepresent the dependency between each related object and the keyword,which gives different weights to each related object, so that differentrelated objects exert different impacts on a recall rate, which bettercomplies with the performance demand of the information retrieval system32, thereby improving the accuracy of the recall rate computing.

The correctness percentage may be computed according to the number ofthe related objects and the number of the non-related objects in theevaluation retrieval result, which is a parameter reflecting theaccuracy and precision of the evaluation retrieval.

In a preferable implementation manner of this embodiment, after thecorrectness percentage of each keyword is computed, an arithmetic meanvalue of the correctness percentages of all keywords is computed.Specifically, the arithmetic mean value of the correctness percentagesis computed according to the following formula:

${P = {\frac{1}{n}{\sum\limits_{k = 1}^{n}\; P_{k}}}},{P_{k} = {\frac{1}{H_{k}}{\sum\limits_{r = 1}^{H_{k}}\; \frac{H_{k} - I_{k,r}}{H_{k}}}}}$

Where, n is the number of keywords in the evaluation keyword set, Pk isthe correctness percentage of a Kth evaluation keyword, Hk is the numberof the related objects included in the evaluation retrieval result ofthe Kth evaluation keyword, and Ik,r is the number of the non-relatedobjects whose arrangement position is in front of the arrangementposition of an rth related object in the evaluation retrieval result ofthe Kth evaluation keyword.

In the foregoing implementation manner, the correctness percentage ofthe evaluation retrieval is computed by adding up the correctnesspercentage of each keyword and obtaining a mean value so that thedistribution condition of a related result is also considered when theamount is considered, thereby improving the computing precision of thecorrectness percentage.

Step 140: The evaluation electronic device 31 computes the evaluationindicator of the information retrieval system 32 according to the recallrate and the correctness percentage.

In this step, the evaluation indicator is computed according to thecomputed recall rate and correctness percentage. In a preferableimplementation manner of this embodiment, the evaluation indicator iscomputed by using an arithmetic mean value R of the recall rates and anarithmetic mean value P of the correctness percentages. Specifically,the evaluation indicator F is computed according to the followingformula:

$F = \frac{\left( {\beta^{2} + 1} \right){PR}}{{\beta^{2}P} + R}$

Where, F is the evaluation indicator, R is an arithmetic mean value ofrecall rates corresponding to all evaluation keywords, P is anarithmetic mean value of correctness percentages corresponding to allevaluation keywords, and β is a pre-determined weight coefficient. Theweight coefficient β is used to adjust an influence weight of the recallrate and the correctness percentage on the evaluation indicator, whereβ<1 indicates emphasizing the accuracy, β>1 indicates emphasizing therecall rate, and β=1 indicates that the weight of the accuracy and theweight of the recall rate are equal.

Definitely, a person skilled in the art may understand that, theevaluation indicator may also be computed through manners such ascomputing a weighted mean value of the recall rate and the correctnesspercentage of each keyword.

In the present disclosure, a ratio between the sum of all the relatedobject parameters of a keyword in keyword in an evaluation retrievalresult set and the sum of all the related object parameters of thekeyword in keyword in a retrieval result set is used to compute a recallrate of an information retrieval system. And the recall rate isintroduced to evaluation the information retrieval system, therebyenhancing accuracy of quantitative evaluation of the informationretrieval system, and improving the automation degree of evaluation.

In the embodiments of the present disclosure, an information retrievalsystem evaluation device, comprising: one or more processors, memory andone or more program units stored in the memory and to be executed by theone or more processors. FIG. 2 is a schematic structural diagram of oneor more program units of an information retrieval system evaluationdevice in a second embodiment of the present disclosure. As shown inFIG. 2, the one or more program units include a behavior data collectingunit 21, an analyzing unit 22, an evaluation retrieval unit 23 and anevaluation indicator computing unit 24.

The behavior data collecting unit 21 is configured to obtain behaviordata sample reported by an information retrieval system within apre-determined period.

Specifically, a related parameter of a user using the informationretrieval system and reported by an interaction module, a searchingmodule and any another module related to subsequent operations of theinformation retrieval system within a pre-determined period (forexample, one month preferably or a longer period) includes a keywordused for retrieval within a certain pre-determined period, an retrievalresult, an arrangement position of each object in the retrieval resultlisted in the retrieval result, and an operation such as whether a userclicks the retrieval result.

In the present disclosure, the retrieval result refers to a retrievedobject list obtained through the information retrieval system, and theobject refers to specific retrieval target of the information retrievalsystem, and may be a web page, a document, a subject, an image, or filesof other type.

The analyzing unit 22 is configured to obtain a sample retrieval keywordset and a sample retrieval result corresponding to each sample retrievalkeyword according to the behavior data sample.

The evaluation retrieval unit 23 is configured to invoke the informationretrieval system to perform evaluation retrieval on a pre-determinedevaluation keyword set, compute a recall rate and a correctnesspercentage corresponding to each keyword in the evaluation keyword setaccording to an evaluation retrieval result and the sample retrievalresult.

The evaluation indicator computing unit 24 is configured to compute anevaluation indicator of the information retrieval system according tothe recall rate and the correctness percentage.

In a preferred implementation manner of this embodiment, the evaluationindicator computing unit 24 computes the evaluation indicator accordingto the following formula.

$F = \frac{\left( {\beta^{2} + 1} \right){PR}}{{\beta^{2}P} + R}$

Where, F is the evaluation indicator, R is an arithmetic mean value ofrecall rates corresponding to all evaluation keywords, P is anarithmetic mean value of correctness percentages corresponding to allevaluation keywords, and β is a pre-determined weight coefficient. Theweight coefficient β is used to adjust an influence weight from therecall rate and the correctness percentage to the evaluation indicator,where β<1 indicates emphasizing the accuracy, β>1 represents emphasizingthe recall rate, and β=1 indicates that the weight of the accuracy andthe weight of the recall rate are equal.

Definitely, a person skilled in the art may understand that, theevaluation indicator may also be computed through manners such ascomputing a weighted mean value of the recall rate and the correctnesspercentage of each keyword.

In a preferred implementation manner of this embodiment, the evaluationretrieval unit 23 includes a recall rate computing subunit 231.

The recall rate computing subunit 231 is configured to compute anarithmetic mean value of the recall rates according to the followingformula:

${R = {\frac{1}{n}{\sum\limits_{k = 1}^{n}\; R_{k}}}},{R_{k} = \frac{{HitScore}_{k}}{{TotalScore}_{k}}}$

Where, n is the number of keywords in the evaluation keyword set, R_(k)is a recall rate of a Kth evaluation keyword, HitScore_(k) is the sum ofall the related object parameters in an evaluation retrieval resultcorresponding to a Kth evaluation keyword, and TotalScore_(k) is the sumof all the related object parameters in a sample retrieval resultcorresponding to a Kth evaluation keyword, wherein a related objectparameter corresponding to the Kth evaluation keyword is computedaccording to the number of times that a user performs operation on arelated object corresponding to the Kth evaluation keyword and anarrangement position of a related object corresponding to the Kthevaluation keyword in its each corresponding sample retrievalsub-result. The related object parameter is used to represent thedependency degree between the related object and the keyword

The related object parameter is computed by the analyzing unit 22,specifically, is computed by the related object parameter subunit 221 ofthe analyzing unit 22.

The related object parameter subunit 221 is configured to compute acorresponding related object parameter score according to the followingformula:

${score} = {{\frac{a}{ExposeCnt}{\sum\limits_{x = 1}^{FCnt}\; {FirstScore}_{x}}} + {\frac{b}{ExposeCnt}{\sum\limits_{y = 1}^{SCnt}\; {SecondScore}_{y}}} + \sigma}$

Where, ExposeCnt is the total number of times that the related objectappeas in a sample retrieval result of a corresponding keyword, FCnt isthe total number of times that a user performs first operation on therelated object appearing in a sample retrieval result of a correspondingkeyword, SCnt is the total number of times that a user performs secondoperation on the related object appearing in a sample retrieval resultof a corresponding keyword, a is a first operation weight coefficient, bis a second operation weight coefficient, σ is an offset coefficient;FirstScorex is a dependency coefficient when the related object appearsin the sample retrieval result of the corresponding keyword for an xthtime and being performed with the first operation, and is computedaccording to the arrangement position of the related object in thesample retrieval sub-result at the retrieval; SecondScorey is adependency coefficient that the related object appears in sampleretrieval result of a corresponding keyword for a yth time and beingperformed with the second operation, and is computed according to thearrangement position of the related object of sample retrievalsub-result at the retrieval.

In this implementation manner, the first operation and the secondoperation may be different operations. For example, when the retrievedobject is a document or a web page, the first operation is click andcheck, the second operation is downloading; when the retrieved object isa music file, the first operation is clicking a listening link, and thesecond operation is downloading; when the retrieved object is a subject,the first operation may be a check link, and the second operation isclicking an order link or performing ordering operation.

In this embodiment, when dependency between the related object and acorresponding keyword is computed, considering the contribution ofseveral different subsequent operations to the dependency, severallatitude modeling is performed, which complies with the demand of aninformation retrieval system of a non-addressing retrieval type.

In a preferred implementation manner of this embodiment, the evaluationretrieval unit 23 further includes a correctness percentage computingsubunit 232.

The correctness percentage computing subunit 232 is configured tocompute the arithmetic mean value of the correctness percentages of thekeyword according to the following formula:

${P = {\frac{1}{n}{\sum\limits_{k = 1}^{n}\; P_{k}}}},{P_{k} = {\frac{1}{H_{k}}{\sum\limits_{r = 1}^{H_{k}}\; \frac{H_{k} - I_{k,r}}{H_{k}}}}}$

Where, n is the number of keywords in the evaluation keyword set, Pk isthe correctness percentage of a Kth evaluation keyword, Hk is the numberof the related objects included in the evaluation retrieval result ofthe Kth evaluation keyword, Ik,r is The number of the non-relatedobjects whose arrangement position is in front of the arrangementposition of an rth related object in the evaluation retrieval result ofthe Kth evaluation keyword.

In this implementation manner, the correctness percentage of theevaluation retrieval is computed through adding the correctnesspercentage of each keyword for mean value seeking so that thedistribution condition of a relevant result is also considered when theamount is considered, thereby improving the computation precision of thecorrectness percentage to a certain degree.

Obviously, a person skilled in the art should be understood that, eachmodule or each step of the embodiments of the present disclosure may beimplemented through a universal computation device, which may beintegrated in a single computation, or distributed in a network systemformed by multiple computation devices. Optionally, they can beimplemented by using an executable code of a computer program, so thatthey can be stored in the storage device to be performed by thecomputing device, or they may be made into each integrated circuitmodule respectively, or they may be implemented by making multiplemodules or steps among them into a single integrated circuit module. Inthis way, the present disclosure is not limited to any specifiedcombination of hardware and software.

In addition, an embodiment of the present disclosure further discloses astorage medium containing a computer executable instruction, thecomputer executable instruction is used to perform an informationretrieval system evaluation method when executed by a compute processor,where the method includes the following steps:

obtaining behavior data sample reported by an information retrievalsystem within a pre-determined period;

obtaining a sample retrieval keyword set and a sample retrieval resultcorresponding to each sample retrieval keyword according to the behaviordata sample;

invoking the information retrieval system to perform evaluationretrieval on a pre-determined evaluation keyword set, computing a recallrate and a correctness percentage corresponding to each keyword in theevaluation keyword set according to an evaluation retrieval result andthe sample retrieval result; and

computing an evaluation indicator of the information retrieval systemaccording to the recall rate and the correctness percentage, wherein

the recall rate is a ratio between the sum of all related objectparameters in the evaluation retrieval result corresponding to a keywordand the sum of all the related object parameters in the sample retrievalresult corresponding to the keyword;

the correctness percentage is computed according to the number of therelated objects and the number of the non-related objects in theevaluation retrieval result corresponding to the keyword; and

the related object corresponding to the keyword is an object where auser performs operation in the sample retrieval result corresponding tothe keyword; the non-related object corresponding to the keyword is anobject where a user does not performs operation in the sample retrievalresult corresponding to the keyword.

The foregoing descriptions are merely preferable embodiments of thepresent disclosure, but are not intended to limit the protection scopeof the present disclosure. For a person skilled in the art, the presentdisclosure may be amended and changed. Any equivalent modification orreplacement readily within the spirit and the principle of the presentdisclosure shall fall within the protection scope of the presentdisclosure.

What is claimed is:
 1. An information retrieval system evaluation method, comprising: obtaining behavior data sample reported by an information retrieval system within a pre-determined period; obtaining a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample; invoking the information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result; and computing an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, where the recall rate is a ratio between the sum of all related object parameters in the evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in the sample retrieval result corresponding to the keyword; the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword; and the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword; the non-related object corresponding to the keyword is an object where a user does not perform operation in the sample retrieval result corresponding to the keyword.
 2. The information retrieval system evaluation method according to claim 1, wherein the evaluation indicator of the information retrieval system is computed according to the following formula: $F = \frac{\left( {\beta^{2} + 1} \right){PR}}{{\beta^{2}P} + R}$ wherein, F is the evaluation indicator, R is an arithmetic mean value of recall rates corresponding to all evaluation keywords, P is an arithmetic mean value of correctness percentages corresponding to all evaluation keywords, and β is a pre-determined weight coefficient.
 3. The information retrieval system evaluation method according to claim 2, wherein the arithmetic mean value of the recall rates is computed according to the following formula: ${R = {\frac{1}{n}{\sum\limits_{k = 1}^{n}\; R_{k}}}},{R_{k} = \frac{{HitScore}_{k}}{{TotalScore}_{k}}}$ wherein, n is the number of keywords in the evaluation keyword set, R_(k) is a recall rate of a Kth evaluation keyword, HitScore_(k) is the sum of all the related object parameters in an evaluation retrieval result corresponding to a Kth evaluation keyword, and TotalScore_(k) is the sum of all the related object parameters in a sample retrieval result corresponding to a Kth evaluation keyword, wherein a related object parameter corresponding to the Kth evaluation keyword is computed according to the number of times that a user performs operation on a related object corresponding to the Kth evaluation keyword and an arrangement position of a related object corresponding to the Kth evaluation keyword in its each corresponding sample retrieval sub-result.
 4. The information retrieval system evaluation method to claim 3, wherein the related object parameter score of each related object is computed according to the following formula: ${score} = {{\frac{a}{ExposeCnt}{\sum\limits_{x = 1}^{FCnt}\; {FirstScore}_{x}}} + {\frac{b}{ExposeCnt}{\sum\limits_{y = 1}^{SCnt}\; {SecondScore}_{y}}} + \sigma}$ wherein, ExposeCnt is the total number of times that the related object appears in a sample retrieval result of a corresponding keyword, FCnt is the total number of times that a user performs first operation on the related object appearing in a sample retrieval result of a corresponding keyword, SCnt is the total number of times that a user performs second operation on the related object appearing in a sample retrieval result of a corresponding keyword, a is a first operation weight coefficient, b is a second operation weight coefficient, σ is an offset coefficient; FirstScore_(x) is a dependency coefficient when the related object appears in the sample retrieval result of the corresponding keyword for an xth time and being performed with the first operation, the dependency coefficient is computed according to arrangement position of the related object in a sample retrieval sub-result in the retrieval; and SecondScore_(y) is a dependency coefficient when the related object appears in the sample retrieval result of the corresponding keyword for a yth time and being performed with the second operation, wherein the dependency coefficient is computed according to arrangement position of the related object in a sample retrieval sub-result in the retrieval.
 5. The information retrieval system evaluation method according to claim 2, wherein the arithmetic mean value of the correctness percentages is computed according to the following formula: ${P = {\frac{1}{n}{\sum\limits_{k = 1}^{n}\; P_{k}}}},{P_{k} = {\frac{1}{H_{k}}{\sum\limits_{r = 1}^{H_{k}}\; \frac{H_{k} - I_{k,r}}{H_{k}}}}}$ wherein, n is the number of keywords in the evaluation keyword set, Pk is the correctness percentage of a Kth evaluation keyword, Hk is the number of the related objects included in the evaluation retrieval result of the Kth evaluation keyword, Ik,r is the number of the non-related objects whose arrangement position is in front of the arrangement position of an rth related object in the evaluation retrieval result of the Kth evaluation keyword.
 6. An information retrieval system evaluation device, comprising: one or more processors, memory and one or more program units stored in the memory and to be executed by the one or more processors, the one or more program units comprising: a behavior data collecting unit, an analyzing unit, an evaluation retrieval unit and an evaluation indicator computing unit; wherein the behavior data collecting unit is configured to obtain behavior data sample reported by an information retrieval system within a pre-determined period; the analyzing unit is configured to obtain a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample; the evaluation retrieval unit is configured to invoke the information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result; and the evaluation indicator computing unit is configured to compute an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, wherein the recall rate is a ratio between the sum of all related object parameters in the evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in the sample retrieval result corresponding to the keyword; the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword; and the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword; the non-related object corresponding to the keyword is an object where a user does not perform operation in the sample retrieval result corresponding to the keyword.
 7. The information retrieval system evaluation device according to claim 6, wherein, the evaluation indicator computing unit is configured to compute the evaluation indicator of the information retrieval system according to the following formula: $F = \frac{\left( {\beta^{2} + 1} \right){PR}}{{\beta^{2}P} + R}$ wherein, F is the evaluation indicator, R is an arithmetic mean value of recall rates corresponding to all evaluation keywords, P is an arithmetic mean value of correctness percentages corresponding to all evaluation keywords, and β is a pre-determined weight coefficient.
 8. The information retrieval system evaluation device according to claim 7, wherein the evaluation retrieval unit comprises a recall rate computing subunit; wherein the recall rate computing subunit is configured to compute an arithmetic mean value of the recall rates according to the following formula: ${R = {\frac{1}{n}{\sum\limits_{k = 1}^{n}\; R_{k}}}},{R_{k} = \frac{{HitScore}_{k}}{{TotalScore}_{k}}}$ wherein, n is the number of keywords in the evaluation keyword set, R_(k) is a recall rate of a Kth evaluation keyword, HitScore_(k) is the sum of all the related object parameters in an evaluation retrieval result corresponding to a Kth evaluation keyword, and TotalScore_(k) is the sum of all the related object parameters in a sample retrieval result corresponding to a Kth evaluation keyword, wherein a related object parameter corresponding to the Kth evaluation keyword is computed according to the number of times that a user performs operation on a related object corresponding to the Kth evaluation keyword and an arrangement position of a related object corresponding to the Kth evaluation keyword in its each corresponding sample retrieval sub-result.
 9. The information retrieval system evaluation device according to claim 8, wherein the analyzing unit comprises a related object parameter computing subunit; wherein the related object parameter subunit is configured to compute a corresponding related object parameter score according to the following formula: ${score} = {{\frac{a}{ExposeCnt}{\sum\limits_{x = 1}^{FCnt}\; {FirstScore}_{x}}} + {\frac{b}{ExposeCnt}{\sum\limits_{y = 1}^{SCnt}\; {SecondScore}_{y}}} + \sigma}$ wherein, ExposeCnt is the total number of times that the related object appears in a sample retrieval result of a corresponding keyword, FCnt is the total number of times that a user performs first operation on the related object appears in a sample retrieval result of a corresponding keyword, SCnt is the total number of times that a user performs second operation on the related object appearing in a sample retrieval result of a corresponding keyword, a is a first operation weight coefficient, b is a second operation weight coefficient, σ is an offset coefficient; FirstScore_(x) is a dependency coefficient when the related object appearing in the sample retrieval result of the corresponding keyword for an xth time and being performed with the first operation, the dependency coefficient is computed according to arrangement position of the related object in a sample retrieval sub-result in the retrieval; and SecondScore_(y) is a dependency coefficient when the related object appears in the sample retrieval result of the corresponding keyword for a yth time and being performed with the second operation, wherein the dependency coefficient is computed according to arrangement position of the related object in a sample retrieval sub-result in the retrieval.
 10. The information retrieval system evaluation device according to claim 7, wherein the evaluation retrieval unit comprises a correctness percentage computing subunit; wherein the correctness percentage computing subunit is configured to compute the arithmetic mean value of the correctness percentages according to the following formula: ${P = {\frac{1}{n}{\sum\limits_{k = 1}^{n}\; P_{k}}}},{P_{k} = {\frac{1}{H_{k}}{\sum\limits_{r = 1}^{H_{k}}\; \frac{H_{k} - I_{k,r}}{H_{k}}}}}$ wherein, n is the number of keywords in the evaluation keyword set, P_(k) is the correctness percentage of a Kth evaluation keyword, H_(k) is the number of the related objects included in the evaluation retrieval result of the Kth evaluation keyword, and I_(k,r) is the number of the non-related objects whose arrangement position is in front of the arrangement position of an rth related object in the evaluation retrieval result of the Kth evaluation keyword.
 11. An storage medium containing a computer executable instruction, wherein the computer executable instruction is used to perform an information retrieval system evaluation method when executed by a compute processor, and the method comprises the following steps: obtaining behavior data sample reported by an information retrieval system within a pre-determined period; obtaining a sample retrieval keyword set and a sample retrieval result corresponding to each sample retrieval keyword according to the behavior data sample; invoking an information retrieval system to perform evaluation retrieval on a pre-determined evaluation keyword set, computing a recall rate and a correctness percentage corresponding to each keyword in the evaluation keyword set according to an evaluation retrieval result and the sample retrieval result; and computing an evaluation indicator of the information retrieval system according to the recall rate and the correctness percentage, wherein the recall rate is a ratio between the sum of all the related object parameters in an evaluation retrieval result corresponding to a keyword and the sum of all the related object parameters in a sample retrieval result corresponding to the keyword; the correctness percentage is computed according to the number of the related objects and the number of the non-related objects in the evaluation retrieval result corresponding to the keyword; and the related object corresponding to the keyword is an object where a user performs operation in the sample retrieval result corresponding to the keyword; the non-related object corresponding to the keyword is an object where a user not performs operation in the sample retrieval result corresponding to the keyword. 